Misattribution of Claims: Comment on Payne Et Al., 2013

Bar-Anan and Nosek (2012) and Payne et al. (2013) investigated the relationship between awareness and perceived intentions with performance on the Affective Misattribution Procedure (AMP). Payne and colleagues incorrectly described some of Bar-Anan and Nosek’s claims, analysis strategies, and conclusions. This brief comment corrects these errors. Bar-Anan and Nosek (2012; hereafter BN) presented evidence that the psychometric qualities of the Affective Misattribution Procedure (AMP, Payne et al., 2005) as an attitude measure are related to people’s retrospective reports whether they detected the effect of their attitude on the measure, and to their reports whether they intentionally caused the attitude effect during the procedure. People who expressed stronger beliefs that their attitudes influenced their performance in the AMP showed better evidence that the AMP measured their attitudes. Additionally, people who expressed stronger beliefs that they intentionally caused the attitude effect in the AMP showed better evidence that the AMP measured their attitudes. A recent paper by Payne and colleagues (2013; hereafter PEA) reported three experiments investigating the reason for BN’s findings. We have some reservations regarding the conclusiveness of those experiments, but their findings do add useful knowledge about the AMP and its properties. In this brief comment, we address PEA’s inaccurate descriptions of BN’s claims and conclusions. PEA misstated BN’s accounts for the AMP effect PEA stated that BN “concluded that the AMP’s validity depends on intentionally rating the primes” (p. 375). This is false. BN argued that there were at least two viable explanations for their findings – intentional ratings (BN, p. 1205) and illusory intention (BN, p. 1205-1206). PEA called intentional ratings “Bar-Anan and Nosek’s account” (PEA, p. 383) and the illusory intention their own account. Further, PEA suggested that BN made a causal conclusion that the strong psychometrics of the AMP were caused by the intentional ratings (PEA, p. 376). However, BN actually concluded that the data could not definitively distinguish between intentional ratings and illusory intention: The present research found that a reliable attitude influence in the AMP is usually accompanied by a perception of intentionally causing this effect. It would be very interesting if this finding is the result of an illusory perception of intentionally causing an unintentional effect. However, because the AMP is gaining much popularity as a measurement tool in the field of attitude research, the alternative possibility—that it is heavily influenced by intentional primes rating— cannot be ignored. Further research might clarify whether perception of intentional primes rating stems from an actual or a retrospectively constructed intention.” (BN, p. 1206). PEA mischaracterized BN’s use of retrospective reports as validity evidence PEA criticized BN by writing that: If the confabulation account is true, then reports of intent are illusory and they result from validly measured preferences (otherwise they would not be associated with actual priming effect sizes). Thus, individual differences in illusory reports of intent cannot provide evidence about whose attitudes are validly measured and whose are not. Retrospective reports of intent could only provide diagnostic evidence about validity if they had accurately reflected the causal processes producing priming effects. (PEA, p. 384) However, BN did not use the retrospective reports as evidence regarding the AMP’s validity or reliability. Rather, BN used the AMP’s effect magnitude, internal consistency, and relations with other attitude measures. BN observed a positive relationship between the validity of the AMP using these criteria and two self-reports: (1) people’s retrospective reports of the priming effect itself and (2) reported intentional rating of the primes. The retrospective report measured the variable “retrospective reports”, not the AMP’s validity. PEA misperceived the definitiveness of BN’s Study 2 In Study 2, BN induced attitudes toward two fictional characters and manipulated the extremity of the preferences between them. BN found that stronger preference induction caused stronger reported beliefs of awareness and intention. PEA argued (p. 384) that these results clearly support the illusory intention account. They reasoned that an increase in the induced preference should increase the priming effect, and the increase in the priming effect should increase the reported (illusory) beliefs of intention to rate the primes. This explanation is consistent with the findings. However, the findings can also be explained with an intentional ratings account. For example, stronger attitudes could have increased the participants’ motivation to express their attitudes toward the primes. Alternatively, having extreme attitudes might have increased the perceived difficulty of ignoring them and thus increase the desirability of intentionally rating the primes to simplify task performance. Thus, as BN concluded, the findings cannot distinguish clearly between intentional and illusory intent accounts. PEA misapplied the nonindependence problem to BN’s analysis strategy PEA argued that BN’s analysis strategy made an error of statistical inference because of nonindependent observations. They described BN’s method this way: First, take any experimental effect, E (i.e., a difference between two experimental conditions), that has systematic variability across persons. Second, find a variable, V, that is positively correlated with the size of the experimental effect (for any reason, spurious or meaningful). Third, divide the sample into subjects who are high versus low on V, and select the subgroup that is low. Finally, interpret the experimental effect among the low-V subgroup. It is bound to be small because it has been selected to be so. All else equal, the stronger the correlation between E and V, the weaker the experimental effect will appear among the low-V group. Substantive interpretations of the experimental effect in this group will be biased to conclude that the experimental effect E is invalid. (pp. 384-5). Adapted to BN’s studies, E is the AMP’s priming effect, V is the self-reported intention, and the low-V subsample is the subsample of people who reported no intentional rating of the primes. BN found that this subsample (41%-62% of the whole sample in BN’s studies) showed hardly any evidence that the AMP reliably measured their attitudes. However, BN did not then conclude that the priming effect is invalid as stated in PEA’s quote. Rather, BN concluded that people who report no intentional rating of the primes hardly show any evidence that the AMP reliably measured their attitudes – a direct description of what was observed. PEA further argued that this inference suffers from the same problem popularly discussed as “voodoo correlations” (Vul, Harris, Winkielman, & Pashler, 2009). That problem occurs by testing many correlations, and then focusing on those that happened to be significantly different from zero. This leads to overestimation of effects. PEA suggested that BN’s “initial selection is based on the association between retrospective self-reports and effect size (or internal consistency) in the AMP. Effect sizes and internal consistency are then interpreted within each subgroup, raising the problem of nonindependence.” (p. 385). This description is inaccurate. There was no initial selection; every level of the moderating variable (retrospective reports) was reported. BN did not present the correlation between priming and retrospective reports (E and V) as a separate finding, independent of the weak priming effect (E) found among those who reported no intentional rating of the primes (the low-V group). The weak priming effect in the low-V group was further information about the correlational result. This is akin to finding that gender is related to the priming effect, and then clarifying that relationship by reporting that men show a very strong effect, whereas women hardly showed the effect. As such, the nonindependence problem described by Vul et al. (2009) does not apply to BN’s analysis strategy. Even if it did, there is a simple solution – replication. If a result is inflated because of selective reporting, replication in confirmatory designs eliminates the biasing influence. BN replicated the correlational finding across four studies, and so did PEA, in their first experiment.